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Aims The population with stable coronary artery disease (SCAD) is growing but validated models to guide their clinical man- 

agement are lacking. We developed and validated prognostic models for all-cause mortality and non-fatal myocardial in- 
farction (Ml) or coronary death in SCAD. 

Methods Models were developed in a linked electronic health records cohort of 102 023 SCAD patients from the CALIBER pro- 

and results gramme, with mean follow-up of 4.4 (SD 2.8) years during which 20 817 deaths and 8856 coronary outcomes were 

observed. The Kaplan-Meier 5-year risk was 20.6% (95% CI, 20.3, 20.9) for mortality and 9.7% (95% CI, 9.4, 9.9) for 
non-fatal Ml or coronary death. The predictors in the models were age, sex, CAD diagnosis, deprivation, smoking, hyper- 
tension, diabetes, lipids, heart failure, peripheral arterial disease, atrial fibrillation, stroke, chronic kidney disease, chronic 
pulmonary disease, liver disease, cancer, depression, anxiety, heart rate, creatinine, white cell count, and haemoglobin. 
The models had good calibration and discrimination in internal (external) validation with C-index 0.811 (0.735) for 
all-cause mortality and 0.778 (0.718) for non-fatal Ml or coronary death. Using these models to identify patients at 
high risk (defined by guidelines as 3% annual mortality) and support a management decision associated with hazard 
ratio 0.8 could save an additional 13-16 life years or 15-18 coronary event-free years per 1000 patients screened, com- 
pared with models with just age, sex, and deprivation. 

Conclusion These validated prognostic models could be used in clinical practice to support risk stratification as recommended in 

clinical guidelines. 
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Introduction 

Population ageing and improvements in survival after acute coronary 
syndromes (ACS) have contributed to worldwide increases in the 
number of patients with stable coronary artery disease (SCAD). 
Stable coronary artery disease encompasses a heterogeneous spec- 
trum of syndromes including patients with stable angina and those 
who have become stable after ACS. 1,2 In the USA alone, over 16 



million people (7% of the US population) suffer from coronary 
disease with 500 000 new stable angina cases being reported each 
year. 3 The 2012 ACCF/AHA guidelines for prognostication in 
SCAD recommend that patients are stratified into high (>3%), inter- 
mediate (1 -3%), and low (< 1%) annual mortality riskgroups, which 
then define different investigation and treatment pathways. 4 

So far, prognostic models proposed for SCAD (compared in Sup- 
plementary material online, Table S1) have been based on data 
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collected for research purposes rather than the information that clini- 
cians record in real-world practice, and none has been recommended 
for clinical use. 4-6 Among their limitations is the failure to incorporate 
the 'broad range of relevant data' identified as important in the guide- 
lines, such as sociodemographic characteristics, cardiovascular (CVD) 
and non-CVD comorbidities, mental health, symptom severity, and 
clinically available biomarkers. 4 These data are routinely recorded for 
most patients before more costly information becomes available 
from further, often invasive, investigations. Other limitations of previous 
models include the use of selected samples from trials 7,8 or voluntary 
registries, 9 and covering a narrow range of SCAD such as excluding 9 
or being confined to post-ACS patients. 7 Furthermore, the ACCF/ 
AHA guidelines emphasize the importance of both all-cause mortality 
and coronary events as outcomes, but no previous studies have 
assessed both. Importantly, none of the previous models has been 
validated in an external data set. 

We sought to address these limitations by analysing large-scale, 
population-based, linked electronic health records data. Our object- 
ive was to develop and validate the performance of prognostic 
models that incorporate clinical measures recommended in guide- 
lines and are commonly available in patients with SCAD. 10 Following 
recent methodological guidance, 1 ' we assessed the accuracy of the 
predictions from the prognostic models based on their calibration, 
discrimination (C-index), and reclassification improvement. 13 To 
further evaluate potential clinical benefits, we estimated life years 
saved when the models are used to guide management decisions. 14 
Based on these analyses, we developed prognostic models to 
predict all-cause mortality and non-fatal myocardial infarction (Ml) 
or coronary death in SCAD and evaluated their performance in an in- 
dependent data set. 

Methods 

Patient population 

We report findings from the CALIBER (CArdiovascular disease research 
using Linked BEspoke studies and Electronic Health Records) collabor- 
ation where we linked population-based primary care data from the 



Clinical Practice Research Datalink (CPRD) to three further sources of 
electronic health records: the Myocardial Ischaemia National Audit 
Project registry (Ml NAP), discharge records from Hospital Episodes Sta- 
tistics (HES), and cause-specific mortality from the Office for National 
Statistics (ONS), as previously described. 15 Eligible patients were those 
with a diagnosis of stable angina, patients with history of Ml, coronary 
artery bypass graft (CABG), or percutaneous coronary intervention 
(PCI) prior to the start of the study period (other CHD) and patients 
with a diagnosis of ACS within the study period (unstable angina or 
acute Ml). Myocardial infarction was classified into ST-elevation Ml 
(STEMI) and non-ST-elevation Ml (NSTEMI) where Ml type was 
recorded (Figure 1). Diagnoses were identified in CPRD, HES, or 
MINAP records according to definitions in the CALIBER data manual. 15 
Stable angina was defined by Read codes in CPRD for angina diagnosis, 
positive ischaemia tests, coronary angiogram results recorded or 
repeat prescriptions for nitrates, or hospitalizations with a primary 
spell diagnosis ICD10 code 120.1, I20.8, or I20.9. Unstable angina was 
defined by Read codes in CPRD or hospital admission with ICD10 
code I20.0. ST-elevation Ml and NSTEMI were defined according to 
the discharge diagnosis as recorded in MINAP. Acute Ml not otherwise 
specified was defined by Read codes in CPRD or ICD10 121-122 as the 
primary diagnosis in HES. Further details on the diagnostic codes and defi- 
nitions used are available at http://www.caliberresearch.org/portal/. 

Patients with prior ACS were defined as stable if they had survived 
more than 6 months following the acute event, entering the cohort at 
this point. We chose 6 months (i) to differentiate long-term prognosis 
from the high-risk period that typically follows ACS or revascularization 1 6 
and (ii) because models validated for clinical use following ACS cover the 
first 6 months post-ACS (e.g. GRACE 17 ). 

Prognostic factors 

Candidate predictors were drawn based on recommendations in recent 
guidelines for the management of SCAD. 4 ' 5 We included demographic 
measures (age, sex, ethnicity, social deprivation), SCAD subtype 
(stable angina, other CHD, unstable angina, Ml, STEMI, NSTEMI), use 
of short- and long-acting nitrates, whether CABG or PCI was performed 
in the 6 months following CAD diagnosis, previous Ml, smoking, body 
mass index (BMI), blood pressure, diagnosis of hypertension, diabetes, 
lipids, family history of coronary heart disease, CVD comorbidities 
[heart failure, peripheral arterial disease (PAD), atrial fibrillation, 
stroke], non-CVD comorbidities (chronic renal disease, chronic 
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obstructive pulmonary disease, thyroid disorders, peptic ulcer, rheuma- 
toid arthritis, cancer, chronic liver disease), psychosocial characteristics 
(depression, anxiety), and clinically assessed biomarkers (heart rate, 
white cell count, haemoglobin, creatinine, liver enzymes, HbA1c). We 
defined as baseline the most recent measurements encompassing 
those made up to 6 months prior to cohort entry. 

Endpoints 

The endpoints were all-cause mortality and a composite of non-fatal Ml 
or coronary death. Patients were censored at the earliest date among 
date of endpoint of interest, relocation to a new primary care practice, 
or study end date (25 March 2010). 

Model development 

Prognostic models were developed and evaluated followingthe checklist 
outlined in the Transparent Reporting of a model for Individual Prognosis 
Or Diagnosis (TRIPOD) guidelines. 18 The baseline hazard for both out- 
comes followed an exponential distribution; hence, we developed expo- 
nential proportional hazards models. These are similarto the Cox model 
in that they assume proportional hazards but the baseline hazard is para- 
metrically estimated based on the exponential model. Log of hazard vs. 
log of time plots showed no significant violations of the proportional 
hazards assumption. Nonlinear associations were modelled using 
splines. Multiple imputation was used to replace missing values in prog- 
nostic factors that appeared to have a missing-at-random pattern 
(details in Supplementary material online). All candidate models a priori 
included age, sex, SCAD subtype, use of long-acting nitrates, CABG/ 
PCI in the last 6 months, previous/recurrent Ml, and CVD risk factors 
(smoking, hypertension, diabetes, total, and HDL cholesterol). Addition- 
al predictors were selected based on their recording coverage and multi- 
variate effect size. Because our data set is large, to decide which variables 
to include in the models we defined as 'moderately significant' those with 
P-values < 0.001 , and 'highly significant' those with P-values < 1 0~ 5 . Thus, 
we examined the multiply imputed multivariate associations of candidate 
predictors, adjusted for all other candidate variables (see Supplementary 
material online, Tables S4 and S5). Predictors with nonlinear associations 
(e.g. BMI) were modelled using restricted cubic splines (three knot points 
were sufficient). Predictors with moderate significance after imputation 
in the multivariate context were retained in the models if the missing 
data did not exceed 50%. Predictors with high significance after imput- 
ation were included if their coverage was at least 20%. Subsequently, 
we examined interactions between age and sex with each of the modelled 
predictors. For simplicity, only the age-sex interaction which was highly 
significant was included in the final models. 

Evaluation of prediction performance 

Discrimination was assessed based on Harrell'sC-index. 19 The contribu- 
tion of individual prognostic factors to the C-index of the full model was 
assessed by backward elimination of a different variable at each iteration. 
The ACCF/AHA guidelines classify low- and high-risk patients as those 
with <1 and >3% annual mortality, respectively. 4 We extended these 
cut-offs to a 5-year time horizon (low <5%, intermediate 5-14%, high 
> 15%) to assess the net reclassification improvement (NRI) 13 upon add- 
ition of different prognostic factors to the models. Calibration of 5-year 
risk predictions was visually assessed by comparing predicted vs. 
observed (Kaplan-Meier) risk by splitting the data into 10 subgroups 
with equal numbers of patients. 

Estimation of life years saved 

We compared the likely clinical impact of using our prognostic models to 
guide medical decisions by estimating the incremental number of life years 



saved by using these models over alternative models, as previously 
described. 14 Briefly, suppose that among a cohort of size N a prognostic 
model identified n patients as high risk. Suppose that upon follow-up the 
observed survival in these patients was S(t). If a risk management decision 
associated with benefit (hazard ratio) 0 was applied to high-risk patients, 
the anticipated risk reduction overtime t would be S(t) 0 -S(t). Estimation 
of the number of life years saved by using the prognostic model takes 
into account the number of patients identified as high risk by the model 
and the expected benefit and cost (medical or other) incurred overtime 
t if specific management decisions were applied to high-risk patients. For 
the current analysis, we defined as high-risk patients those with >15% 
5-year risk (20% in sensitivity analysis). We assumed that the management 
decision (e.g. furthertesting ortreatments) is associated with a hazard ratio 
of 0.8 (~20% risk reduction) [for context, antiplatelet therapy has been 
shown to reduce risk of fatal/non-fatal CVD by ~20% in patients with 
prior Ml or confirmed CAD 20 ]. Further, we assumed that the cost asso- 
ciated with implementing the decision balances the benefit obtained in 
patients with 15% 5-year risk (i.e. treating people with higher or lower 
observed risk results in positive or negative net benefit, respectively). 

Validation 

We validated models internally (in CALIBER) and externally (different 
study and clinical setting). 21 Within CALIBER, we estimated bootstrap 
standard errors (200 samples) to obtain optimism-corrected confidence 
intervals. 22 For external validation, the models developed in CALIBER 
were applied to 4020 patients in the Appropriateness of Coronary 
Revascularisation' (ACRE) study, a prospective cohort of patients, with 
similar case-mix to patients in our data, who underwent coronary angiog- 
raphy at the London Chest Hospital during 1996/1997 and followed up 
until 2004. Details of this study have been previously described. 23 Covari- 
ates in ACRE that were incompletely recorded (smoking, TCHOL, HDL, 
heart rate, blood pressure, creatinine, haemoglobin, white cell count) 
were multiply imputed as described for the development data set, but in- 
dependently of the CALIBER data. The following candidate predictors 
considered in our model development had not been collected by the 
ACRE investigators: history of anxiety, depression, cancer, liver disease, 
and atrial fibrillation (all binary). To allow models containing these vari- 
ables to be evaluated in ACRE, we set these to 0. Discrimination and cali- 
bration statistics for 5-year risks predictions in the external data set were 
assessed as in the main analysis. 

All analyses were performed in the R statistical package, version 2.15.2 
for Unix. 

Results 

Baseline patient characteristics in different 
forms of stable coronary artery disease 

We included 102 023 patients, of whom 44.7% had stable angina, 
21.5% other CHD, 13.0% unstable angina, 4.7% STEMI, 6.7% 
NSTEMI, and 9.4% had unclassified Ml as their qualifying diagnosis. 
Compared with other patients with SCAD, STEMI patients were 
younger, more likely to be male and smokers, more likely to 
receive PCI, had fewer comorbidities and lower SBP and heart rate 
(Supplementary material online, Table S2). 

Five-year risks of all-cause mortality and 
non-fatal Ml or coronary death 

The Kaplan-Meier 5-year mortality risk was 20.6% (95% CI, 20.3, 
20.9%) but differed markedly among different SCAD subtypes 
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(Figure 2). The lowest 5-year mortality was observed in stable angina 
patients and those stable after STEM I and the highest in stable patients 
after NSTEMI. The observed 5-year risk of non-fatal Ml or coronary 
death was approximately half that of all-cause mortality [9.7% (95% 
CI, 9.4, 9.9%)], but more heterogeneous among SCAD subtypes, 
with the lowest seen in patients with stable angina and the highest 
in stable patients after NSTEMI. 

Prognostic factors 

The age- and sex-adjusted associations of candidate prognostic 
factors with either of the two endpoints were comparable among 
the different SCAD subtypes (Supplementary material online, 
Tables S3 and S4). Male sex was associated with higher risks in all 
forms of SCAD except after Ml, whereas age was a stronger risk 
factor in women. Body mass index had a U shape association with all- 
cause mortality and a positive linear association with coronary out- 
comes, in both cases changing to L shape after multiple adjustments 
(Supplementary material online, Table S5). Because of the potential 
for over-adjustment when diabetes, hypertension, and other factors 
strongly associated with obesity were included, 24 BMI was not 
included in the final models. Hyperthyroidism and high HbA1c had 
significant univariate but weak multivariate associations. All continu- 
ous predictors appearing in the final models were approximately lin- 
early associated with the corresponding outcomes. 

The CALIBER models 

We developed models for all-cause mortality and non-fatal Ml or cor- 
onary death that include the following prognostic factors (different 
coefficients for each endpoint);age, sex, deprivation, SCAD 
subtype, recent revascularizations, previous Ml, use of long-acting 
nitrates, diabetes, hypertension, smoking, lipids, heart failure, PAD, 
atrial fibrillation, stroke, chronic renal disease, chronic obstructive 
pulmonary disease, cancer, chronic liver disease, depression, 
anxiety, heart rate, creatinine, white cell count, and haemoglobin 
(Table J). 



Discrimination and calibration 

In internal validation, the C-index was 0.81 1 (95% CI, 0.806, 0.81 6) for 
all-cause mortality and 0.778 (95% CI, 0.770, 0.785) for non-fatal Ml 
or coronary death. For interpretation, a model with C-index 0.81 1 
correctly predicts who will have an event earlier between two 
patients 81.1% of the times, a significant improvement over a 
random 50% guess. The models were well calibrated throughout 
the risk range (Supplementary material online, Figure S1). 

Contribution of prognostic factors to 
model performance 

Predictors based on initial CAD diagnosis, CVD co-morbidities, and 
biomarkers contributed the largest increments in the C-index 
(Figure 3). For prediction of all-cause mortality, the predictors with 
the largest influence on the discrimination of the model were (in 
order of decreasing contribution to the C-index): creatinine, total 
and HDL cholesterol, SCAD subtype with revascularization status, 
depression, heart rate, and chronic obstructive pulmonary disease. 
Equivalently for prediction of Ml and coronary death, the most influ- 
ential predictors were SCAD subtype, haemoglobin, heart failure, 
use of nitrates, diabetes, PAD, white cell count, and previous Ml. 
Other non-CVD comorbidities contributed to the prediction of all- 
cause mortality only, whereas diagnosis of hypertension did not lead 
to gain in discrimination performance for either endpoint. 

Evaluation of potential clinical impact 

To evaluate the potential clinical impact of using these models for 
clinical evaluation of patients with SCAD, the full models were com- 
pared with one with just demographic characteristics (Figure 4 and 
Supplementary material online, figure S J). Based on 5-year risk pre- 
dictions, the NRI was 0.35 (95% CI, 0.34, 0.36) for all-cause mortality 
and 0.24 (95% CI, 0.23, 0.26) for non-fatal Ml or coronary death. For 
interpretation, an NRI of 0.35 in the current context means that in a 
population of patients classified into three categories according to 
their 5-year risk predictions (<5, 5-14, and >15%) based on an 




Figure 2 Observed (Kaplan -Meier) risk of all cause-mortality and non-fatal Ml or coronary death for stable angina patients and stable patients 6 
months after ACS. The numbers of patients at risk per unit time are shown below each plot. 
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Table I The CALIBER models for prognosis of all-cause mortality and non-fatal myocardial infarction or coronary death 
in stable coronary artery disease 



Prognostic factors 


AU-cause mortality, 


Non-fatal Ml or coronary death, 




HR a (95% CI) 


HR b (95% CI) 


bociodemographic characteristics 






Age in men, per year 


I .UO J ^ I.UOj - I.UO/ J 


1 n4A m 04^ 1 n4P.i 


Age in women, per year 


1 Oft1 (\ ("I7ft 1 WX\ 


I.UOJ ^ I.UO I — I.UD7J 


Women vs. men 


U.ZUT ^u. lOZ — U.Zj/ J 


U. I JO [y. I iz — U.ZZ I ^ 


Most deprived cjuintile, yes vs. no 


11 MM 111 11 Ql\ 
I . I J I \ \-\ I l — I.I 7 A) 


1 117/1 OAn 1 1 7P.1 
I.I I / ^ I .uou — I . I / 0^ 


SCAD diagnosis and severity 






Other CHD vs. stable angina 


1.024 (0.982-1.067) 


1.180 (1.107-1.257) 


Unstable angina vs. stable angina 


1 mi fd 970 1 07^"\ 
I .UZ I yJ.y i\J — \.\JiJj 


1 "39"? /1 939 1 4911 
I.JZJ ^ I.ZJZ — I j 


M'sTFMI \/c ctTklo ^no-ins 
IN j 1 CI II Vb. ildLHc angina 


1 99ft f1 1 ^AfYl 
I.Z.70 I I.ZjO — I.jdUI 


9 ?73 H 917 9 ^391 

Z.J/J 1 Z.Z 1 / — Z.JJ/I 


SXEMI vs. stable angina 


1 oft? c\ nnA 1 1 AfVi 

1 .UOJ 1 1 .UuD — 1 . 1 DDI 


1 940 (1 7^fl 9 149*1 

1 . 7^U I l./JU — Z. IT/I 


P( in act H rnnntnc vp>c \/c nn 

1 V — . 1 Ml Lajl U 1 1 1 1 1 L.I 1 0 , yCj Vj. \ \\J 


0 651 CO 605-0 699) 


0 70? CO 642-0 7681 


( ARfi in lact A mi^ntnc vp>c \/c nn 

^w/AUVJ III la^L KJ 1 1 1 KJ 1 1 LI 1 o . ycj VJ. 1 1 \J 


0 516 (0469-0 5661 


0 494 CO 370-0 4861 


Previous/recurrent Ml, yes vs. no 


1 09^ 1 1791 
I.IjDI I.U7J 1.1/71 


1 "399 M ?^fl 1 4791 

I.J77 I I.JJU — 1 .TV x. J 


Use of nitrates, yes vs. no 


1.152 (1.118-1.188) 


1.405 (1.342-1.470) 


\—vlj nsk Tactors 






Ex-smoker vs. never 


1 nn/i cia^ 1 1 ^7\ 

I.I I U ^ I .UOJ— I . I J / ) 


1 094 /1 nnp. 1 1 ftft^ 

I.U77 ^ I.UUO— I. \O0) 


fi ir"r*^nt cmn/^r \/c n^v/f^r* 
v Ul ICIIL illlU l*«t: 1 V3. 1 1 1: V C 1 


1 "31 S (1 74S — 1 ^ft91 

1 J IJ 1 1 . Z.~J 1 JO/I 


1 71 5 H 17n— 1 "31 *h 

I .Z. I J t I . 1 Z.O 1 . J 1 II 


Hypertension, present vs. absent 


n 9^^ r*n 999 1 nm ,■ 

U.7DJ yJ.y A" — I .UU I J 


1 riA4 /1 nnn i m\ 

I .Uo7 ^ I .uuu— I . I J I J 


Diabetes mellitus, present vs. absent 


1.203 (1.160-1.248) 


1.387(1.315-1.463) 


Total cholesterol, per 1 mmol/L increase 


1.012(0.983-1.042) 


1.061 (1.029-1.094) 


HDL, per 0.5 mmol/L increase 


1.006 (0.987-1.025) 


0.910(0.879-0.942) 


. 

v_VI_J co-morbidities 






Heart failure, present vs. absent 


1 ^4? M 4QI; 1 
I.J1J I i.i7j — 1 .J/J 1 


1 1P.1 /1 0P9 1 9P11 

1. IO 1 I I.U07 — I.ZO 1 1 


Peripheral arterial disease, present vs. absent 


1.286(1.234-1.340) 


1.085(1.031-1.142) 


Atrial fibrillation, present vs. absent 


1.280(1.236-1.326) 


0.952 (0.887-1.021) 


Stroke, present vs. absent 


1.329 (1.277-1.382) 


1.138 (0.925-1.401) 


Non-CVD comorbidities 






Chronic kidney disease, present vs. absent 


1.116(1.058-1.178) 


1.085(1.031-1.142) 


Chronic obstructive pulmonary disease, present vs. absent 


1.150(1.114-1.187) 


1.181 (1.089-1.281) 


Cancer, present vs. absent 


1.377(1.324-1.432) 


0.952 (0.887-1.021) 


Chronic liver disease, present vs. absent 


1.631 (1.443-1.842) 


1.138 (0.925-1.401) 


Psychosocial characteristics 






Depression at diagnosis, present vs. absent 


1.179 (1.135-1.225) 


1.059 (0.998-1.124) 


Anxiety at diagnosis, present vs. absent 


1.172(1.116-1.231) 


1.015 (0.937-1.100) 


Biomarkers 






Heart rate, per 10 b.p.m increase 


1.098(1.084-1.112) 


1.069 (1.030-1.110) 


Creatinine, per 30 fxmoUL increase 


1.065 (1.051-1.080) 


1.087 (1.064-1.110) 


White cell count, per 1.5 10 9 /L increase 


1.120(1.106-1.135) 


1.111 (1.088-1.135) 


Haemoglobin, per 1.5 g/dL increase 


0.758 (0.724-0.794) 


0.822 (0.800-0.845) 


C-index 






Internal cross-validation 


0.811 (0.806-0.816) 


0.778 (0.770-0.785) 


External validation (ACRE) 


0.735 (0.715-0.755) 


0.718(0.700-0.736) 



CABG, coronary artery bypass graft; SCAD, stable coronary artery disease; HDL, high-density lipoprotein cholesterol; Ml, myocardial infarction; NSTEMI, non-ST-segment elevation 
Ml; PAD peripheral arterial disease; PCI, percutaneous coronary intervention; STEMI, ST-segment elevation Ml. 
Exponential proportional hazards regression with scale 92 (95% CI, 89, 94). 
Exponential proportional hazards regression with scale 88 (95% CI, 84, 92). 
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All-cause mortality 



1. SOCIODEMOGRAPHICS 

- deprivation 

2. CAD DIAGNOSIS & SEVERITY 

- CAD subtype, PCI/CABG 

- recurrent Ml 

- nitrates use 

3. PRIMARY CVD RISK FACTORS 

- smoking status 

- hypertension 

- diabetes 

- TCHOL. HDL 

4. CVD COMORBIDITIES 

- heart failure 

- peripheral arterial disease 

- atrial fibrillation 

- hist- of stroke 

5. NON-CVD COMORBIDITIES 

- chronic renal disease 

- chronic pulmonary disease 

- cancer 

- chronic liver disease 

6. PSYCHOSOCIAL 

- depression 

- anxiety 

7. BIOMARKERS 

- heart rate 

- creatinine 

- white cell count 

- haemoglobin 



Diff. C-index 95% CI 

-0.0005 I-0.0007: -0.0002] 



-0.0030 [-0.0040: 
-0.0006 [-0.0009, 
-0.0007 1-0.0011 



-0.0021] 
-0.0003] 
-0.0003] 



0.0000 1-0.0001; 0.0001] 

-0.0003 [-0.0005: 0.0000] 

0.0000 [0.0000; 0.0000] 

-0.0042 [-0.0049; -0.0035] 



-0.0010 [-0.0013; 

-0.0011 [-0.0015; 

-0.0011 [-0.0014: 

-0.0O02 [-0.0003. 



-0.0004 [-0.0006, 

-0.0018 [-0.0023; 

-0.0003 [-0.0005. 

-0.0009 [-0.0013: 



-0.0007] 
-0.0007] 
-0.0008] 
-0.0001] 



-0.0003] 
-0.0013] 
-0.0001] 
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Figure 3 The contribution of each variable included in the CALIBER model for predicting all-cause mortality in SCAD, assessed by the decrease in 
the C-index upon its removal from the complete model. 



existing model, updating the model by including further risk factors 
correctly reclassifies 35% of the patients into risk groups that are 
more concordant with their observed event rates, i.e. patients are 
moved to the lower risk category if their observed event rate was 
low and vice versa. If specific management strategies were recom- 
mended for high-risk patients (>15% 5-year risk), and implementa- 
tion of these management strategies was associated with hazard 
ratio 0.8, using the CALIBER models to estimate 5-year risk would 
save an additional 14.5 (95% CI, 13.0, 16.0) life years or 16.1 (95% 
CI, 14.6, 17.6) non-fatal Ml or coronary death-free years for every 
1000 patients screened. Using a 20% threshold to define high-risk 
patients led to similar conclusions. 

External validation 

For external validation, we applied the CALIBER models to patients 
in the ACRE study, summarized in Supplementary material online, 
Table S6. During a median follow-up of 7.5 years, 872 deaths and 
1238 non-fatal Ml or coronary deaths were observed among 4020 
patients. The C-index was 0.735 (95% CI, 0.71 5, 0.755) for all-cause 
mortality and 0.71 8 (95% CI, 0.700, 0.736) for non-fatal Ml or coron- 
ary death. Five-year risks were well calibrated (Supplementary 
material online, Figure S1). In sensitivity analysis, we built and evalu- 
ated models in CALIBER that omitted the predictors missing in 
ACRE (history of depression, anxiety, caner, liver disease, and 
atrial fibrillation); the corresponding performance was very similar 
to that observed for the full models (where missing predictors 
were set to 0). 



Risk calculation example 

The models proposed are available online at www.caliberresearch. 
org/model. Scoring tables 25 to simplify the estimation of the risk 
scores offline are provided in Supplementary material online. 
Suppose we wish to assess the prognosis of a 65-year old man, 
who recently survived a STEMI and has remained stable for the last 
6 months. He is a current smoker, suffers from diabetes, and has fre- 
quent bouts of angina, which he controls using nitrates. He has no 
other history of CVD but suffers with depression and chronic ob- 
structive pulmonary disease. His pulse rate is 80 b.p.m. and his 
current blood measurements are: total cholesterol 7 mmol/L, HDL 
cholesterol 2 mmol/L, creatinine 100 mmol/L, white blood count 
10 x 10 9 /L, and haemoglobin 14g/L. No revascularization proced- 
ure was performed following his heart attack. From the scoring 
tables provided, we have total score for death model = 146 , corre- 
sponding to ~ 1 2.5% 5-year riskof death; total score for M l/fatal CH D 
model = 202, corresponding to ~6% 5-year risk of Ml/coronary 
death. According to the guidelines, patients with annual risk of 
death or Ml >3% (> 15% risk over 5 years) are candidates for revas- 
cularization. This patient's mortality risk is near this threshold, so he 
should be assessed further (e.g. ECG, angiography) to confirm his 
suitability for revascularization. 

Discussion 

In a population-based study of > 100 000 stable patients with a range 
of previous CAD phenotypes, we found that prognostic models 
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Characteristics used in risk-assessment 
(incrementally updated models) 



1. SOCIODEMOGRAPHICS 

age, sex, multiple deprivation 

2. CAD DIAGNOSIS & SEVERITY 

CAD subtype, PCI/CABG. hist Ml. nitrate use 

3. PRIMARY CVD RISK FACTORS 

smoking, hypertension, diabetes, lipids 

4. CVD CO-MORBIDITIES 

heart failure. PAD. atrial fibrillation, stroke 

5. NON-CVD COMORBIDITIES 

chronic renal disease. COPD, cancer, liver disease 

6. PSYCHOSOCIAL 

depression, anxiety 

7. BIOMARKERS 

heart rate, creatinine, white cell count, haemoglobin 



Difference in the C-index 



95% CI C-lndex 



Diff in 
C-index 

0.000 [0.000; 0.000] 0.754 



B 

Non-fatal A 



T — l — l — l 

0 0.01 0.03 
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0.013 [0.011; 0.015] 0.767 

0.005 [0.004; 0.006] 0.772 

0.017 [0.016; 0.018] 0.790 

0.006 [0 005; 0.007] 0.795 
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0.014 [0.011:0.017] 0.811 



Clinical Net Reclassification Index* 

NRI 95% CI 

0.000 [0 000; 0 000] 

0.105 [0.096:0.114] 
0.076 [0.069; 0.082] 
0.122 [0.111; 0.134] 
0 054 [0.050; 0 059] 
0.027 (0.023; 0.031] 
10.162 [0.152; 0.173] 



t r 

0 0.050.10.15 
proportion correctly re-classified 



Life-yeans gained if models used 
to support management decision" 

Life-years 95% CI 

0.0 [0.0; 0.0] 

2.8 [2 0: 3.7] 
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4 9 [3.9: 6 0] 
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0.4 [0.1:0.7] 
3.2 [2.3:4.1] 



1 1 1 1 

0 2 4 6 8 
Life-years gained/1000 screened 



Characteristics used in risk-assessment 
(incrementally updated models) 



1. SOCIODEMOGRAPHICS 

age, sex, multiple deprivation 

2. CAD DIAGNOSIS & SEVERITY 

CAO subtype. PCI/CABG. hist. Ml, nitrate use 

3. PRIMARY CVD RISK FACTORS 

smoking, hypertension, diabetes, lipids 

4 CVD CO-MORBIDITIES 

heart failure. PAD, atrial fibrillation, stroke 

5. NON-CVD COMORBIDITIES 

chronic renal disease, COPD. cancer, liver disei 

6. PSYCHOSOCIAL 

depression, anxiety 

7. BIOMARKERS 

heart rate, creatinine, white cell count, haemoglobin 



T 1 1 1 

0 0.01 0.03 
change in proportion of concordant pairs 



Difference in the C-index 

„ Dmin 95% CI C-lndex 
C-index 

0.000 [0.000; 0.0O0] 0.703 

- 0.042 [0.037; 0.047] 0.744 

0 009 [0 007; 0.01 1] 0.753 

0.014 [0.013: 0.016] 0 768 

0.001 [0.001; 0.002] 0.769 

0.000 [0 000: 0.000] 0.769 

0 009 [0.006; 0.012] 0.778 



Clinical Net Reclassification Index' 

NRI 95% CI 

0.000 I 0.000; 0.000] 



0.102 [0.089:0.115] 
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0.048 1 0 030:0 066] 
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1 1 1 

0 0.050.10.15 
proportion correcuy re-classified 



Life-years gained if models used 
to support management decision" 

Life-years 95% CI 

0.0 [0.0:0.0] 

7.5 [6.6:8.4] 
1.5 [0.8:2.1] 
3.3 [2.2:4.4] 
0.7 [0.3:1.0] 
0.1 [-0.2:0.3] 
2.5 [1.2:3.8] 



~1 1 1 1 
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Figure 4 Evaluation of prediction performance upon incremental addition of different sets of predictors, using sociodemographics as the refer- 
ence model. *Based on guideline definition of low (<1%), intermediate (1 -3%), and high (>3%) annual risk. **Assumingthat management is asso- 
ciated with hazard ratio 0.8 and is cost-effective at high risk. 



combining a wide range of clinical data commonly available prior to 
the decision for further investigation can identify patients at high 
risk of long-term mortality and coronary events. We demonstrate 
how real-world clinical data (as distinct from research data 7,8 ) con- 
tribute important prognostic information in unselected patients. 
Unlike previous reports of prognostic models, 1,7-9 we focus on 
potential clinical usefulness and demonstrate how each predictor 
usefully improves predictions beyond more simple models. Import- 
antly, we confirm that the models have good calibration and discrim- 
ination when applied to an external study. These validated prognostic 
models could be used clinically to support risk assessment according 
to clinical variables, which is 'essential for determining optimal treat- 
ment strategies'. 4 

Current practice guidelines informed several aspects of this ana- 
lysis. 4,5 Because risks in SCAD are heterogeneous and many patients 
have low risk, the 2012 ACCF/AHA 4 and 2006 ESC 5 guidelines 
propose a stepwise risk assessment to inform management decisions. 
The assessment should start with evaluating all patients based on 
simple clinical parameters, followed by advanced investigations 
(imaging and stress testing) in selected patients, followed by angiog- 
raphy conditional on results from earlier steps. The CALIBER 



prognostic models incorporate the broad range of clinical character- 
istics highlighted by these guidelines for the initial evaluation step, 
many of which (deprivation, atrial fibrillation, cancer, liver disease, de- 
pression, anxiety, and haemoglobin) have not previously been incor- 
porated in prognostic models for SCAD. Importantly, to make the 
evaluation of the models relevant to the guidelines, we used the 
ACCF/AHA definition of low, intermediate, and high riskto estimate 
the NRI and life years gained from applying these models. Hence, our 
evaluation outcomes are clinically relevant. 

We present both all-cause mortality and non-fatal Ml or coronary 
death as outcomes because these risks differ and the best course of 
action in terms of risk management and priorities requires evaluation 
of both mortality and coronary morbidity. Some prognostic factors 
included in the CALIBER models were much more useful for predict- 
ing all-cause mortality than coronary risks and vice versa. Hence, al- 
though smoking, diabetes, CVD comorbidities, and biomarkers were 
good predictors of either outcome, non-CVD comorbidities, de- 
pression, and anxiety were mainly useful for predicting non-coronary 
death. Importantly, the contribution of hypertension to model dis- 
crimination was marginal and confined to coronary events. Since 
blood pressure lowering trials among people with SCAD 26 
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demonstrate a reduction in both coronary and total mortality rates, 
the prognostic effect of hypertension may be obscured by treatment. 

Widely reported measures of model performance, such as the 
C-indexand the NRI, do not provide clinicians with readily interpret- 
able evidence for making effective, and cost-effective decisions about 
which patients should be further investigated. 27 Following recom- 
mendations, 10 we therefore sought to estimate the potential 
impact on patient outcomes of using the CALIBER prognostic 
models for clinical decision making. Using these models to identify 
patients at high risk (defined by guidelines as 3% annual mortality) 
and offer a management strategy with a hazard ratio 0.8 (equivalent 
to ~20% absolute risk reduction) would save an additional 13-16 
life years or 15-18 coronary-event-free life years over a 5-year 
time horizon compared with a model including just age, sex, and 
social deprivation. Hence, since the models are based on clinically 
available data (i.e. there is no cost increment of collecting new 
data), our analysis demonstrates that screening with the models is 
likely to be cost-effective. However, economic evaluation would be 
needed to establish their appropriateness in practice. 

Stringent external validation is essential to ensure that a prognostic 
model would be applicable to the rest of the population than the 
patients in our cohort. 21 However, a model can achieve impressive 
performance (as high as in the development data) if applied to an ex- 
ternal data set that originates from a similar source of data (e.g. valid- 
ation of QRISK 28 in data from a similar GP database system 29 ). To 
make our external validation rigorous, we therefore used a data set 
with major differences from the development data set; the data 
were collected differently (manually abstracted from case records), 
the patients were at higher risk (received angiography due to chest 
pain), and the study period was about a decade earlier with markedly 
different background use of risk lowering medication. Hence, it was 
reassuring that the models performed well in this external setting. 
However, as with any prognostic model, there is an ongoing need 
for further validation in external data sets and recalibration for differ- 
ent populations and time periods. 

We propose that the CALIBER prognostic models are implemented 
in electronic health records in clinical practice. This should be done 
alongside evaluation of the impact on decisions and, ultimately, 
patient outcomes. The models may also be used to support adherence 
to existing therapies, guide frequency of follow-up, plan trials of new 
therapies, and serve as a reference on which to evaluate the extent to 
which novel biomarkers might usefully further stratify risk assessment. 

Limitations 

A limitation in our study is thatthe electronic health record does not 
capture all the information available to clinicians prior to imaging. 
Thus, while clinicians assess symptom severity, this is not recorded 
in standard fashion. However, we found that a proxy for symptom se- 
verity, the use of long-acting nitrates, or repeat prescription for short 
nitrates did add prognostic value. Although we had information from 
the resting ECG on heart rate and the presence of atrial fibrillation, 
resting ST segment, T wave, or other changes were seldom coded. 
A further limitation was that for external evaluation, we ignored 
(assumed absent) comorbidities not collected in the external data 
set. However, the missing predictors had low impact on the perform- 
ance of the models in the development data; thus, we do not expect 
this to have affected the evaluation. Further, in order to represent all 



relevant clinical parameters that are commonly considered in clinical 
assessment, the models included a large number of variables. 
However, these variables are routinely recorded in electronic 
health records, so their collection is not associated with extra 
costs. Also, the large size of the data set used to develop the 
models reduces the likelihood of overfitting. Finally, only multiplica- 
tive models were considered; models fitted in the additive scale are a 
plausible alternative that was not explored. 

Conclusions 

In summary, we present validated prognostic models for estimating 
risk of all-cause mortality and coronary outcomes based on clinical 
parameters that are commonly available in all people with stable 
coronary disease. These models can be implemented alongside 
further medical investigations to support medical decision making. 
However, as with any new prognostic model, further independent 
evaluation is required in different settings including different elec- 
tronic health record systems, health care organizations, and geo- 
graphic locations to guide use in clinical practice. A risk calculator is 
available online (www.caliberresearch.org/model). 

Ethics and registration 

We registered the research protocol at clinical trials.gov 
(NCT01 609465) and received approval from the I ndependent Scien- 
tific Advisory Committee (10_1 60) and the MINAP Academic Group 
(11-CBR-3). 

Supplementary material 

Supplementary material is available at European Heart Journal online. 

Funding 

This work was supported by the National Institute for Health Research 
(RP-PG-0407-10314), the Wellcome Trust (WT 08609 1/Z/08/Z), and 
the Medical Research Council Prognosis Research Strategy Partnership 
(G0902393/99558) [to H.H., AT., P.P., A.D.H.], and by awards to estab- 
lish the Farr Institute of Health Informatics Research at UCLPartners, 
from the Medical Research Council, Arthritis Research UK, British 
Heart Foundation, Cancer Research UK, Chief Scientist Office, Economic 
and Social Research Council, Engineering and Physical Sciences Research 
Council, National Institute for Health Research, National Institute for 
Social Care and Health Research, and Wellcome Trust (AT., S.D., 
A.D.H., L.S., H.H.,grant MR/K006584/1). L.S. is supported by a Wellcome 
Trust Senior Research Fellowship in Clinical Science. AT. acknowledges 
support of Barts and the London Cardiovascular Biomedical Research 
Unit, funded by the National Institute for Health Research. A.S. is 
supported by a Wellcome Trust Clinical Research Training Fellowship 
(0938/30/Z/10/Z). J.G. was funded by an NIHR Doctoral Fellowship 
(DRF-2009-02-50). Funding to pay the Open Access publication 
charges for this article was provided by Wellcome Trust and the 
Medical Research Council. 

Conflict of interest: none declared. 

References 

1. Bhatt DL, Eagle KA, Ohman EM, Hirsch AT, Goto S, Mahoney EM, Wilson PW, 
Alberts MJ, D'Agostino R, Liau CS, Mas JL, Rother J, Smith SC Jr, Salette G, 
Contant CF, MassaroJM, Steg PG. Comparative determinants of 4-year cardiovascu- 
lar event rates in stable outpatients at risk of or with atherothrombosisJAMA 2010; 
304:1350-1357. 



852 



E. Rapsomaniki et ai 



2. Hemingway H, McCallum A, Shipley M, Manderbacka K, Martikainen P, Keskimaki I. 
Incidence and prognostic implications of stable angina pectoris among women and 
men. JAMA 2006;295:1404-1411. 

3. Roger VL, Go AS, Lloyd-Jones DM, Benjamin EJ, BerryJD, Borden WB, Bravata DM, 
Dai S, Ford ES, Fox CS, Fullerton HJ, Gillespie C, Hailpern SM, Heit JA, Howard VJ, 
Kissela BM, Kittner SJ, Lackland DT, Lichtman JH, Lisabeth LD, Makuc DM, 
Marcus GM, Marelli A, Matchar DB, Moy CS, Mozaffarian D, Mussolino ME, 
Nichol G, Paynter NP, Soliman EZ, Sorlie PD, Sotoodehnia N, Turan TN, 
Virani SS, Wong ND, Woo D, Turner MB. Executive summary: heart disease and 
stroke statistics — 2012 update: a report from the American Heart Association. Gr- 
culation 2012;125:188-197. 

4. Fihn SD, Gardin JM, Abrams J, Berra K, Blankenship JC, Dallas AP, Douglas PS, 
Foody JM, Gerber TC, Hinderliter AL, King SB III, Kligfield PD, Krumholz HM, 
Kwong RY, Lim MJ, Linderbaum JA, Mack MJ, Munger MA, Prager RL, Sabik JF, 
Shaw LJ, Sikkema JD, Smith CR Jr, Smith SC Jr, Spertus JA, Williams SV. 2012 
ACCF/AHA/ACP/AATS/PCNA/SCAI/STS guideline for the diagnosis and manage- 
ment of patients with stable ischemic heart disease: executive summary: a report 
of the American College of Cardiology Foundation/American Heart Association 
task force on practice guidelines, and the American College of Physicians, American 
Association for Thoracic Surgery, Preventive Cardiovascular Nurses Association, 
Society for Cardiovascular Angiography and Interventions, and Society of Thoracic 
Surgeons. Circulation 2012;126:3097-3137. 

5. Fox K, Garcia MA, Ardissino D, Buszman P, Camici PG, Crea F, Daly C, de Backer G, 
Hjemdahl P, Lopez-Sendon j, Marco J, MoraisJ, Pepper J, Sechtem U, Simoons M, 
Thygesen K, Priori SG, Blanc JJ, Budaj A, Camm J, Dean V, Deckers J, Dickstein K, 
Lekakis J, McGregor K, Metra M, Morais J, Osterspey A, Tamargo J, Zamorano JL. 
Guidelines on the management of stable angina pectoris: executive summary: The 
Task Force on the Management of Stable Angina Pectoris of the European Society 
of Cardiology. Eur Heart J 2006;27:1 341 -1381. 

6. Henderson RA, O'Flynn N. Management of stable angina: summary of NICE guid- 
ance. Heart 201 2;98:500-507. 

7. Marschner IC, Colquhoun D, Simes RJ, Glasziou P, Harris P, Singh BB, Friedlander D, 
White H, Thompson P, Tonkin A. Long-term risk stratification for survivors of acute 
coronary syndromes. Results from the Long-term Intervention with Pravastatin in 
Ischemic Disease (LIPID) Study. LIPID Study Investigators. J Am Coll Cardiol 2001; 
38:56-63. 

8. Clayton TC, Lubsen J, Pocock Sj, Voko Z, Kirwan BA, Fox KA, Poole-Wilson PA. 
Risk score for predicting death, myocardial infarction, and stroke in patients with 
stable angina, based on a large randomised trial cohort of patients. BMJ 2005;331: 
869. 

9. Daly CA, De Stavota B, Sendon JL, Tavazzi L, Boersma E, Clemens F, Danchin N, 
Delahaye F, Gitt A, Julian D, Mulcahy D, Ruzyllo W, Thygesen K, Verheugt F, 
Fox KM. Predicting prognosis in stable angina — results from the Euro heart survey 
of stable angina: prospective observational study. BMJ 2006;332:262-267. 

10. Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MS, Go AS, 
Harrell FEJr, Hong Y, Howard BV, Howard VJ, Hsue PY, Kramer CM, McConnellJP, 
Normand SL, O'Donnell CJ, Smith SCJr, Wilson PW. Criteria for evaluation of novel 
markers of cardiovascular risk: a scientific statement from the American Heart As- 
sociation. Circulation 2009;119:2408-2416. 

11. Henriksson M, Palmer S, Chen R, DamantJ, FitzpatrickNK, Abrams K, Hingorani AD, 
Stenestrand U, Janzon M, Feder G, Keogh B, Shipley MJ, Kaski JC, Timmis A, 
Sculpher M, Hemingway H. Assessing the cost effectiveness of using prognostic bio- 
markers with decision models: case study in prioritising patients waiting for coronary 
artery surgery. BMJ 2010;340:5606. 

12. Steyerberg EW, Vickers AJ, Cook NR. Gerds T, Gonen M, Obuchowski N, 
Pencina MJ, Kattan MW. Assessing the performance of prediction models: a frame- 
work for traditional and novel measures. Epidemiology 2010;21:128-138. 



1 3. Pencina MJ, D'Agostino RB Sr, Steyerberg EW. Extensions of net reclassification im- 
provement calculations to measure usefulness of newbiomarkers. Stat Med 201 1;30: 
11-21. 

14. Rapsomaniki E, White IR, Wood AM, Thompson SG. A framework for quantifying 
net benefits of alternative prognostic models. Stat Med 2012;31:1 14-1 30. 

15. Denaxas SC, George J, Herrett E, Shah AD, Kalra D, Hingorani AD, Kivimaki M, 
Timmis AD, Smeeth L, Hemingway H. Data resource profile: cardiovascular 
disease research using linked bespoke studies and electronic health records 
(CALIBER). IntJ Epidemiol 2012;41:1625-1638. 

16. Hirsch A, Verouden NJ, Koch KT, BaanJJr, HenriquesJP, PiekJJ, RohlingWJ, van der 
Schaaf RJ,TijssenJG, Vis MM, de Winter RJ. Comparison of long-term mortality after 
percutaneous coronary intervention in patients treated for acute ST-elevation myo- 
cardial infarction versus those with unstable and stable angina pectoris. Am J Cardiol 
2009;104:333-337. 

1 7. Fox KA, Dabbous OH, Goldberg RJ, Pieper KS, Eagle KA, van de Werf F, Avezum A, 
Goodman SG, Flather MD, Anderson FAJr, Granger CB. Prediction of risk of death 
and myocardial infarction in the six months after presentation with acute coronary 
syndrome: prospective multinational observational study (GRACE). BMJ 2006;333: 
1091. 

18. Steyerberg EW, Moons KG, van der Windt DA, Hayden JA, Perel P, Schroter S, 
Riley RD, Hemingway H, Altman DG. Prognosis Research Strategy (PROGRESS) 
3: prognostic model research. PLoS Med 201 3;1 0:e1001 381 . 

19. Harrell FEJr, Califf RM, Pryor DB, Lee KL, Rosati RA. Evaluating the yield of medical 
tests. JAMA 1982;247:2543-2546. 

20. Antithrombotic Trialists' Collaboration. Collaborative meta-analysis of randomised 
trials of antiplatelet therapy for prevention of death, myocardial infarction, and 
stroke in high risk patients. BMJ 2002;324:71 -86. 

21. Altman DG, Royston P. What do we mean by validating a prognostic model? StatMed 
2000;19:453-473. 

22. Harrell FEJr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing 
models, evaluating assumptions and adequacy, and measuring and reducing errors. 
Stat Med 1996;15:361-387. 

23. Hemingway H, Crook AM, Feder G, Banerjee S, Dawson JR, Magee P, Philpott S, 
Sanders J, Wood A, Timmis AD. Underuseof coronary revascularization procedures 
in patients considered appropriate candidates for revascularization. N Engl J Med 
2001;344:645-654. 

24. Lewis CE, McTigue KM, Burke LE, Poirier P, Eckel RH, Howard BV, Allison DB, 
Kumanyika S, Pi-Sunyer FX. Mortality, health outcomes, and body mass index in 
the overweight range: a science advisory from the American Heart Association. Gr- 
culation 2009;119:3263-3271. 

25. Sullivan LM, MassaroJM, D'Agostino RBSr. Presentation of multivariate data for clin- 
ical use: the Framingham Study risk score functions. Stat Med 2004;23:1 631 - 1 660. 

26. Law MR, Morris JK.Wald NJ. Use of blood pressure lowering drugs in the prevention 
of cardiovascular disease: meta-analysis of 147 randomised trials in the context of 
expectations from prospective epidemiological studies. BMJ 2009;338:b1 665. 

27. Hemingway H, Croft P, Perel P, Hayden JA, Abrams K, Timmis A, Briggs A, 
Udumyan R, Moons KG, Steyerberg EW, Roberts I, Schroter S, Altman DG, 
Riley RD. Prognosis research strategy (PROGRESS) 1 : a framework for researching 
clinical outcomes. BMJ 201 3;346:e5595. 

28. Hippisley-CoxJ, Coupland C, Vinogradova Y, RobsonJ, Minhas R, Sheikh A, Brindle P. 
Predicting cardiovascular risk in England and Wales: prospective derivation and val- 
idation of QRISK2. BMJ 2008;336:1475-1482. 

29. Collins GS, Altman DG. An independent external validation and evaluation ofQRISK 
cardiovascular risk prediction: a prospective open cohort study. BMJ 2009;339: 
b2584. 



